Using deep learning to track time × frequency whistle contours of toothed whales without human-annotated training data.

Many odontocetes produce whistles that feature characteristic contour shapes in spectrogram representations of their calls. Automatically extracting the time × frequency tracks of whistle contours has numerous subsequent applications, including species classification, identification, and density estimation. Deep-learning-based methods, which train models using analyst-annotated whistles, offer a promising way to reliably extract whistle contours. However, the application of such methods can be limited by the significant amount of time and labor required for analyst annotation. To overcome this challenge, a technique that learns from automatically generated pseudo-labels has been developed. These annotations are less accurate than those generated by human analysts but more cost-effective to generate. It is shown that standard training methods do not learn effective models from these pseudo-labels. An improved loss function designed to compensate for pseudo-label error that significantly increases whistle extraction performance is introduced. The experiments show that the developed technique performs well when trained with pseudo-labels generated by two different algorithms. Models trained with the generated pseudo-labels can extract whistles with an F1-score (the harmonic mean of precision and recall) of 86.31% and 87.2% for the two sets of pseudo-labels that are considered. This performance is competitive with a model trained with 12 539 expert-annotated whistles (F1-score of 87.47%).

Ecological inferences about marine mammals from passive acoustic data.

Monitoring on the basis of sound recordings, or passive acoustic monitoring, can complement or serve as an alternative to real-time visual or aural monitoring of marine mammals and other animals by human observers. Passive acoustic data can support the estimation of common, individual-level ecological metrics, such as presence, detection-weighted occupancy, abundance and density, population viability and structure, and behaviour. Passive acoustic data also can support estimation of some community-level metrics, such as species richness and composition. The feasibility of estimation and certainty of estimates is highly context dependent, and understanding the factors that affect the reliability of measurements is useful for those considering whether to use passive acoustic data. Here, we review basic concepts and methods of passive acoustic sampling in marine systems that often are applicable to marine mammal research and conservation. Our ultimate aim is to facilitate collaboration among ecologists, bioacousticians, and data analysts. Ecological applications of passive acoustics require one to make decisions about sampling design, which in turn requires consideration of sound propagation, sampling of signals, and data storage. One also must make decisions about signal detection and classification and evaluation of the performance of algorithms for these tasks. Investment in the research and development of systems that automate detection and classification, including machine learning, are increasing. Passive acoustic monitoring is more reliable for detection of species presence than for estimation of other species-level metrics. Use of passive acoustic monitoring to distinguish among individual animals remains difficult. However, information about detection probability, vocalisation or cue rate, and relations between vocalisations and the number and behaviour of animals increases the feasibility of estimating abundance or density. Most sensor deployments are fixed in space or are sporadic, making temporal turnover in species composition more tractable to estimate than spatial turnover. Collaborations between acousticians and ecologists are most likely to be successful and rewarding when all partners critically examine and share a fundamental understanding of the target variables, sampling process, and analytical methods.

Silbido profundo: An open source package for the use of deep learning to detect odontocete whistles.

This work presents an open-source matlab software package for exploiting recent advances in extracting tonal signals from large acoustic data sets. A whistle extraction algorithm published by Li, Liu, Palmer, Fleishman, Gillespie, Nosal, Shiu, Klinck, Cholewiak, Helble, and Roch [(2020). Proceedings of the International Joint Conference on Neural Networks, July 19-24, Glasgow, Scotland, p. 10] is incorporated into silbido, an established software package for extraction of cetacean tonal calls. The precision and recall of the new system were over 96% and nearly 80%, respectively, when applied to a whistle extraction task on a challenging two-species subset of a conference-benchmark data set. A second data set was examined to assess whether the algorithm generalized to data that were collected across different recording devices and locations. These data included 487 h of weakly labeled, towed array data collected in the Pacific Ocean on two National Oceanographic and Atmospheric Administration (NOAA) cruises. Labels for these data consisted of regions of toothed whale presence for at least 15 species that were based on visual and acoustic observations and not limited to whistles. Although the lack of per whistle-level annotations prevented measurement of precision and recall, there was strong concurrence of automatic detections and the NOAA annotations, suggesting that the algorithm generalizes well to new data.

A practical guide for generating unsupervised, spectrogram-based latent space representations of animal vocalizations.

BACKGROUND: The manual detection, analysis and classification of animal vocalizations in acoustic recordings is laborious and requires expert knowledge. Hence, there is a need for objective, generalizable methods that detect underlying patterns in these data, categorize sounds into distinct groups and quantify similarities between them. Among all computational methods that have been proposed to accomplish this, neighbourhood-based dimensionality reduction of spectrograms to produce a latent space representation of calls stands out for its conceptual simplicity and effectiveness. Goal of the study/what was done: Using a dataset of manually annotated meerkat Suricata suricatta vocalizations, we demonstrate how this method can be used to obtain meaningful latent space representations that reflect the established taxonomy of call types. We analyse strengths and weaknesses of the proposed approach, give recommendations for its usage and show application examples, such as the classification of ambiguous calls and the detection of mislabelled calls. What this means: All analyses are accompanied by example code to help researchers realize the potential of this method for the study of animal vocalizations.

Performance metrics for marine mammal signal detection and classification.

Automatic algorithms for the detection and classification of sound are essential to the analysis of acoustic datasets with long duration. Metrics are needed to assess the performance characteristics of these algorithms. Four metrics for performance evaluation are discussed here: receiver-operating-characteristic (ROC) curves, detection-error-trade-off (DET) curves, precision-recall (PR) curves, and cost curves. These metrics were applied to the generalized power law detector for blue whale D calls [Helble, Ierley, D'Spain, Roch, and Hildebrand (2012). J. Acoust. Soc. Am. 131(4), 2682-2699] and the click-clustering neural-net algorithm for Cuvier's beaked whale echolocation click detection [Frasier, Roch, Soldevilla, Wiggins, Garrison, and Hildebrand (2017). PLoS Comp. Biol. 13(12), e1005823] using data prepared for the 2015 Detection, Classification, Localization and Density Estimation Workshop. Detection class imbalance, particularly the situation of rare occurrence, is common for long-term passive acoustic monitoring datasets and is a factor in the performance of ROC and DET curves with regard to the impact of false positive detections. PR curves overcome this shortcoming when calculated for individual detections and do not rely on the reporting of true negatives. Cost curves provide additional insight on the effective operating range for the detector based on the a priori probability of occurrence. Use of more than a single metric is helpful in understanding the performance of a detection algorithm.

Introduction to the special issue on machine learning in acoustics.

The use of machine learning (ML) in acoustics has received much attention in the last decade. ML is unique in that it can be applied to all areas of acoustics. ML has transformative potentials as it can extract statistically based new information about events observed in acoustic data. Acoustic data provide scientific and engineering insight ranging from biology and communications to ocean and Earth science. This special issue included 61 papers, illustrating the very diverse applications of ML in acoustics.

Using context to train time-domain echolocation click detectors.

This work demonstrates the effectiveness of using humans in the loop processes for constructing large training sets for machine learning tasks. A corpus of over 57 000 toothed whale echolocation clicks was developed by using a permissive energy-based echolocation detector followed by a machine-assisted quality control process that exploits contextual cues. Subsets of these data were used to train feed forward neural networks that detected over 850 000 echolocation clicks that were validated using the same quality control process. It is shown that this network architecture performs well in a variety of contexts and is evaluated against a withheld data set that was collected nearly five years apart from the development data at a location over 600 km distant. The system was capable of finding echolocation bouts that were missed by human analysts, and the patterns of error in the classifier consist primarily of anthropogenic sources that were not included as counter-training examples. In the absence of such events, typical false positive rates are under ten events per hour even at low thresholds.

Improve automatic detection of animal call sequences with temporal context.

Many animals rely on long-form communication, in the form of songs, for vital functions such as mate attraction and territorial defence. We explored the prospect of improving automatic recognition performance by using the temporal context inherent in song. The ability to accurately detect sequences of calls has implications for conservation and biological studies. We show that the performance of a convolutional neural network (CNN), designed to detect song notes (calls) in short-duration audio segments, can be improved by combining it with a recurrent network designed to process sequences of learned representations from the CNN on a longer time scale. The combined system of independently trained CNN and long short-term memory (LSTM) network models exploits the temporal patterns between song notes. We demonstrate the technique using recordings of fin whale (Balaenoptera physalus) songs, which comprise patterned sequences of characteristic notes. We evaluated several variants of the CNN + LSTM network. Relative to the baseline CNN model, the CNN + LSTM models reduced performance variance, offering a 9-17% increase in area under the precision-recall curve and a 9-18% increase in peak F1-scores. These results show that the inclusion of temporal information may offer a valuable pathway for improving the automatic recognition and transcription of wildlife recordings.

Publisher Correction: Deep neural networks for automated detection of marine mammal species.

An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Deep neural networks for automated detection of marine mammal species.

Deep neural networks have advanced the field of detection and classification and allowed for effective identification of signals in challenging data sets. Numerous time-critical conservation needs may benefit from these methods. We developed and empirically studied a variety of deep neural networks to detect the vocalizations of endangered North Atlantic right whales (Eubalaena glacialis). We compared the performance of these deep architectures to that of traditional detection algorithms for the primary vocalization produced by this species, the upcall. We show that deep-learning architectures are capable of producing false-positive rates that are orders of magnitude lower than alternative algorithms while substantially increasing the ability to detect calls. We demonstrate that a deep neural network trained with recordings from a single geographic region recorded over a span of days is capable of generalizing well to data from multiple years and across the species' range, and that the low false positives make the output of the algorithm amenable to quality control for verification. The deep neural networks we developed are relatively easy to implement with existing software, and may provide new insights applicable to the conservation of endangered species.

Machine learning in acoustics: Theory and applications.

Acoustic data provide scientific and engineering insights in fields ranging from biology and communications to ocean and Earth science. We survey the recent advances and transformative potential of machine learning (ML), including deep learning, in the field of acoustics. ML is a broad family of techniques, which are often based in statistics, for automatically detecting and utilizing patterns in data. Relative to conventional acoustics and signal processing, ML is data-driven. Given sufficient training data, ML can discover complex relationships between features and desired labels or actions, or between features themselves. With large volumes of training data, ML can discover models describing complex acoustic phenomena such as human speech and reverberation. ML in acoustics is rapidly developing with compelling results and significant future promise. We first introduce ML, then highlight ML developments in four acoustics research areas: source localization in speech processing, source localization in ocean acoustics, bioacoustics, and environmental sounds in everyday scenes.

Automated classification of dolphin echolocation click types from the Gulf of Mexico.

Delphinids produce large numbers of short duration, broadband echolocation clicks which may be useful for species classification in passive acoustic monitoring efforts. A challenge in echolocation click classification is to overcome the many sources of variability to recognize underlying patterns across many detections. An automated unsupervised network-based classification method was developed to simulate the approach a human analyst uses when categorizing click types: Clusters of similar clicks were identified by incorporating multiple click characteristics (spectral shape and inter-click interval distributions) to distinguish within-type from between-type variation, and identify distinct, persistent click types. Once click types were established, an algorithm for classifying novel detections using existing clusters was tested. The automated classification method was applied to a dataset of 52 million clicks detected across five monitoring sites over two years in the Gulf of Mexico (GOM). Seven distinct click types were identified, one of which is known to be associated with an acoustically identifiable delphinid (Risso's dolphin) and six of which are not yet identified. All types occurred at multiple monitoring locations, but the relative occurrence of types varied, particularly between continental shelf and slope locations. Automatically-identified click types from autonomous seafloor recorders without verifiable species identification were compared with clicks detected on sea-surface towed hydrophone arrays in the presence of visually identified delphinid species. These comparisons suggest potential species identities for the animals producing some echolocation click types. The network-based classification method presented here is effective for rapid, unsupervised delphinid click classification across large datasets in which the click types may not be known a priori.

Acoustic differentiation of Shiho- and Naisa-type short-finned pilot whales in the Pacific Ocean.

Divergence in acoustic signals used by different populations of marine mammals can be caused by a variety of environmental, hereditary, or social factors, and can indicate isolation between those populations. Two types of genetically and morphologically distinct short-finned pilot whales, called the Naisa- and Shiho-types when first described off Japan, have been identified in the Pacific Ocean. Acoustic differentiation between these types would support their designation as sub-species or species, and improve the understanding of their distribution in areas where genetic samples are difficult to obtain. Calls from two regions representing the two types were analyzed using 24 recordings from Hawai'i (Naisa-type) and 12 recordings from the eastern Pacific Ocean (Shiho-type). Calls from the two types were significantly differentiated in median start frequency, frequency range, and duration, and were significantly differentiated in the cumulative distribution of start frequency, frequency range, and duration. Gaussian mixture models were used to classify calls from the two different regions with 74% accuracy, which was significantly greater than chance. The results of these analyses indicate that the two types are acoustically distinct, which supports the hypothesis that the two types may be separate sub-species.

Acoustic sequences in non-human animals: a tutorial review and prospectus.

Animal acoustic communication often takes the form of complex sequences, made up of multiple distinct acoustic units. Apart from the well-known example of birdsong, other animals such as insects, amphibians, and mammals (including bats, rodents, primates, and cetaceans) also generate complex acoustic sequences. Occasionally, such as with birdsong, the adaptive role of these sequences seems clear (e.g. mate attraction and territorial defence). More often however, researchers have only begun to characterise - let alone understand - the significance and meaning of acoustic sequences. Hypotheses abound, but there is little agreement as to how sequences should be defined and analysed. Our review aims to outline suitable methods for testing these hypotheses, and to describe the major limitations to our current and near-future knowledge on questions of acoustic sequences. This review and prospectus is the result of a collaborative effort between 43 scientists from the fields of animal behaviour, ecology and evolution, signal processing, machine learning, quantitative linguistics, and information theory, who gathered for a 2013 workshop entitled, 'Analysing vocal sequences in animals'. Our goal is to present not just a review of the state of the art, but to propose a methodological framework that summarises what we suggest are the best practices for research in this field, across taxa and across disciplines. We also provide a tutorial-style introduction to some of the most promising algorithmic approaches for analysing sequences. We divide our review into three sections: identifying the distinct units of an acoustic sequence, describing the different ways that information can be contained within a sequence, and analysing the structure of that sequence. Each of these sections is further subdivided to address the key questions and approaches in that area. We propose a uniform, systematic, and comprehensive approach to studying sequences, with the goal of clarifying research terms used in different fields, and facilitating collaboration and comparative studies. Allowing greater interdisciplinary collaboration will facilitate the investigation of many important questions in the evolution of communication and sociality.

Acoustic behavior of melon-headed whales varies on a diel cycle.

Many terrestrial and marine species have a diel activity pattern, and their acoustic signaling follows their current behavioral state. Whistles and echolocation clicks on long-term recordings produced by melon-headed whales (Peponocephala electra) at Palmyra Atoll indicated that these signals were used selectively during different phases of the day, strengthening the idea of nighttime foraging and daytime resting with afternoon socializing for this species. Spectral features of their echolocation clicks changed from day to night, shifting the median center frequency up. Additionally, click received levels increased with increasing ambient noise during both day and night. Ambient noise over a wide frequency band was on average higher at night. The diel adjustment of click features might be a reaction to acoustic masking caused by these nighttime sounds. Similar adaptations have been documented for numerous taxa in response to noise. Or it could be, unrelated, an increase in biosonar source levels and with it a shift in center frequency to enhance detection distances during foraging at night. Call modifications in intensity, directionality, frequency, and duration according to echolocation task are well established for bats. This finding indicates that melon-headed whales have flexibility in their acoustic behavior, and they collectively and repeatedly adapt their signals from day- to nighttime circumstances.

Compensating for the effects of site and equipment variation on delphinid species identification from their echolocation clicks.

A concern for applications of machine learning techniques to bioacoustics is whether or not classifiers learn the categories for which they were trained. Unfortunately, information such as characteristics of specific recording equipment or noise environments can also be learned. This question is examined in the context of identifying delphinid species by their echolocation clicks. To reduce the ambiguity between species classification performance and other confounding factors, species whose clicks can be readily distinguished were used in this study: Pacific white-sided and Risso's dolphins. A subset of data from autonomous acoustic recorders located at seven sites in the Southern California Bight collected between 2006 and 2012 was selected. Cepstral-based features were extracted for each echolocation click and Gaussian mixture models were used to classify groups of 100 clicks. One hundred Monte-Carlo three-fold experiments were conducted to examine classification performance where fold composition was determined by acoustic encounter, recorder characteristics, or recording site. The error rate increased from 6.1% when grouped by acoustic encounter to 18.1%, 46.2%, and 33.2% for grouping by equipment, equipment category, and site, respectively. A noise compensation technique reduced error for these grouping schemes to 2.7%, 4.4%, 6.7%, and 11.4%, respectively, a reduction in error rate of 56%-86%.

Spatio-temporal patterns of beaked whale echolocation signals in the North Pacific.

At least ten species of beaked whales inhabit the North Pacific, but little is known about their abundance, ecology, and behavior, as they are elusive and difficult to distinguish visually at sea. Six of these species produce known species-specific frequency modulated (FM) echolocation pulses: Baird's, Blainville's, Cuvier's, Deraniyagala's, Longman's, and Stejneger's beaked whales. Additionally, one described FM pulse (BWC) from Cross Seamount, Hawai'i, and three unknown FM pulse types (BW40, BW43, BW70) have been identified from almost 11 cumulative years of autonomous recordings at 24 sites throughout the North Pacific. Most sites had a dominant FM pulse type with other types being either absent or limited. There was not a strong seasonal influence on the occurrence of these signals at any site, but longer time series may reveal smaller, consistent fluctuations. Only the species producing BWC signals, detected throughout the Pacific Islands region, consistently showed a diel cycle with nocturnal foraging. By comparing stranding and sighting information with acoustic findings, we hypothesize that BWC signals are produced by ginkgo-toothed beaked whales. BW43 signal encounters were restricted to Southern California and may be produced by Perrin's beaked whale, known only from Californian waters. BW70 signals were detected in the southern Gulf of California, which is prime habitat for Pygmy beaked whales. Hubb's beaked whale may have produced the BW40 signals encountered off central and southern California; however, these signals were also recorded off Pearl and Hermes Reef and Wake Atoll, which are well south of their known range.

Offshore killer whale tracking using multiple hydrophone arrays.

To study delphinid near surface movements and behavior, two L-shaped hydrophone arrays and one vertical hydrophone line array were deployed at shallow depths (<125 m) from the floating instrument platform R/P FLIP, moored northwest of San Clemente Island in the Southern California Bight. A three-dimensional propagation-model based passive acoustic tracking method was developed and used to track a group of five offshore killer whales (Orcinus orca) using their emitted clicks. In addition, killer whale pulsed calls and high-frequency modulated (HFM) signals were localized using other standard techniques. Based on these tracks sound source levels for the killer whales were estimated. The peak to peak source levels for echolocation clicks vary between 170-205 dB re 1 µPa @ 1 m, for HFM calls between 185-193 dB re 1 µPa @ 1 m, and for pulsed calls between 146-158 dB re 1 µPa @ 1 m.

Species-specific beaked whale echolocation signals.

Beaked whale echolocation signals are mostly frequency-modulated (FM) upsweep pulses and appear to be species specific. Evolutionary processes of niche separation may have driven differentiation of beaked whale signals used for spatial orientation and foraging. FM pulses of eight species of beaked whales were identified, as well as five distinct pulse types of unknown species, but presumed to be from beaked whales. Current evidence suggests these five distinct but unidentified FM pulse types are also species-specific and are each produced by a separate species. There may be a relationship between adult body length and center frequency with smaller whales producing higher frequency signals. This could be due to anatomical and physiological restraints or it could be an evolutionary adaption for detection of smaller prey for smaller whales with higher resolution using higher frequencies. The disadvantage of higher frequencies is a shorter detection range. Whales echolocating with the highest frequencies, or broadband, likely lower source level signals also use a higher repetition rate, which might compensate for the shorter detection range. Habitat modeling with acoustic detections should give further insights into how niches and prey may have shaped species-specific FM pulse types.